Semantics Altering Modifications for Evaluating Comprehension in Machine Reading

Advances in NLP have yielded impressive results for the task of machine reading comprehension (MRC), with approaches having been reported to achieve performance comparable that humans. In this paper, we investigate whether state-of-the-art MRC models are able correctly process Semantics Altering Modifications (SAM): linguistically-motivated phenomena alter semantics a sentence while preserving most its lexical surface form. We present method automatically generate and align challenge sets featuring original altered examples. further propose novel evaluation methodology assess capability systems these examples independent data they were optimised on, by discounting effects introduced domain shift. large-scale empirical study, apply order evaluate extractive regard their SAM-enriched data. comprehensively cover 12 different neural architecture configurations four training datasets find -- despite well-known remarkable consistently struggle semantically